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Scope 


This presentation is meant to be a unified vision for EDG’s future 

development and testing strategy 

This presentation summarizes the findings from: 

0 The White Paper: “Internal Review of Current EDG Testing Practices” 

0 The White Paper: “Independent Review of EDG Test Programs” 

0 Recommendations from the Tiger Team looking at EDG tradecraft 
0 Recommendations from recent EDG management offsite 
In addition it will show EDG’s vision for: 

0 The automated test suite, DART 

0 The software development and collaboration tool suite by Atlassian 
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Apologies Up Front 


Reasons the pitchforks are about to come out 
0 Introduces new process 
0 Increase in documentation 
0 More work up front 
0 It is change 

Reasons the pitchforks should go away 
0 Less work in the long run 

0 Assists junior developers in learning EDG tradecraft 
0 Protects our tools 
0 Green check marks are wonderful 
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Vocabulary 


There are a few items we will be mentioning by name often: 
Atlassian Products - (Replaced TeamForge) 

0 Confluence - Knowledge Management 
0 Jim - Project Management and Issue Tracking 
0 Bamboo - Continuous Integration 
0 Stash - Source Control using Git 
Other Products 

0 DART - Automated testing suite replaced ERGOSTAR 
0 Git - Distributed version control system replaced SVN 
0 ServiceNow - Replacement for IMIS 

0 Tool Pedigree database (TPD) - Database being compiled based upon 
recommendations of all Tiger Teams 
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Git should be used for all projects 
Two repositories for each project 
0 One for code (EDG only) 

0 One for tests (shared with COG) 
Two long lived branches 
0 Master - Official releases 
0 Develop - Stable branch 

Three short lived branch types 

0 Feature - development done here 
0 Release - code freeze for testing 
0 Hotfix - immediate fixes to master 







Four Phases of a Project 


The wall between development and 
evaluation should be extremely low 

Migration between the four phases 
should be controlled via Bamboo to 
ensure what code is where 

A memorandum of understanding 
between EDG and its customers must 
be crafted so that customers can be 
trusted with evaluation copies 
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Development 



During this phase the development team is in its most isolated state, 
and is focused purely on the requirements on hand 
Tasks during this phase: 

0 Development of code base (by a team ) 

0 Creation of unit tests 
0 Code reviews 

0 Documentation of capabilities in TPD 
Development during this phase shall: 

0 Be done in feature branches 
0 Be focused on one thing at a time 


AED: Adopt the ' Git Flow ' workflow described previously 

ESD: Recommend contracts follow a similar workflow (or at least vocabulary) 
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The Development Te 



The development team should be broken into three categories 
Development 

0 Lead Developer - Project lead 
0 Additional Developers (always at least one) 

Testing 

0 Lead Tester - Doesn’t answer to Project lead, keeps the Y in IV&V 
0 Additional Testers (if necessary) 

Project Support 
0 Branch Chief 
0 Systems Integrator 


AED: Development will be conducted in teams; testers shall be equal partners in 
development so they can fully understand the operation 
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Unit Testing 



“The primary goal of unit testing is to take the smallest piece of 
testable software in the application, isolate it from the remainder of 
the code, and determine whether it behaves exactly as you expect" 
Unit tests should test: 

° The Good - expected values 
° The Bad - extreme, but acceptable values 
° The Ugly - out of range values 

Unit testing is more work up front, but saves time and produces better 
code 


AED: Developers should incorporate unit testing during development 
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Code Reviews via 
Stash 



Conduct frequent code reviews earlier 
while developing 

What is a code review / pull request? 

° Focus on incorporation of single 
development task into source base 
° Review new changes to source base by 
other team members 
° Use peer review to exercise EDG best 
practices 

° Document new capabilities in TPD 
° Tame the chaos of the merge process 


Pull request 


Feature/PHILO-3 copy included payload to target computer 



AED: Code reviews should be conducted via Stash before merging into ' develop ’ 
ESD: COTRs should request information to fill in the TPD 
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Evaluation 



During this phase the development team interacts with the testing 
team and the customers to ensure the product being created fits with 
the customers’ vision 
Tasks during this phase: 

° Creation of integration test scripts or an integration test plan 
° User and developer guide creation 
° Demonstration of current functionality 
° Refinement of requirements 

Evaluation versions are never deployable. If there is a need, a new 
requirement can be generated 


AED and ESD: Pass evaluation versions to the customer earlier to receive 
feedback often; solicit customer feedback often 
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Issue Tracking via JIF 



Breaking down requirements into 
basic tasks makes time estimation 
more realistic 

° Batch tasks into shorter development 
iterations (Agile) 

° Assign tasks to development team 
members 

° Track discovered bugs with new tasks 
° Capture reported issues from Customer 
Works alongside ServiceNow 
° ServiceNow tracks official requirements 
° Jira tracks work done to meet those 
requirements through to delivery 



AED: Use JIRA during development to track progress 

AED and ESD: Use JIRA as an official way for feedback on evaluation versions 
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Documentation via 
Confluence 



Improve documentation by using 
Confluence 

° Centralize all project knowledge in one 
location (external to developers' brains!) 
° Export Confluence pages to PDF format 
Create new project standards for 
documentation 

° User Guide (formal and informal) 

° Developer Guide (knowledge retention) 

° Tool Pedigree (human readable) 


Philosoraptor Home 




Q User Guide updated yesterday at 2:55 PM • view change 


AED: Collaborate on project documentation, increase project visibility to ESD 
ESD: Import documentation to Confluence, increase project visibility to AED 


UNCLASSIFIED//FOUO 




Quality Assurance 



The decision to cut a release candidate should be a decision between 
the development team, the testing team, and the branch chief 
Questions that need to be answered: 

° Is there sufficient unit test coverage 
° Are there automated integration tests in place 
° Is there a test plan in place 

° What type of regression testing is needed for this version 

Keep in mind that after this step a tool could potentially be deployed 
with no further changes 


AED: Establish a practice of having a sit-down with the project lead, branch 
chief and testing lead before cutting a release candidate 
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Am 

Release Candidate W 


A release candidate signifies a code freeze for all requirements and 
only bug fixes should be introduced to the codebase during this phase 
Tasks during this phase: 

° Tier One testing 

° Acceptance testing by IV&V on bare metal (if required) 

° Basic Forensic Testing 
° Finalization of all documentation 

Operationally deployable only with EDG COPs approval 


AED and ESD: Establish a M OU stating that evaluation versions are never to be 
deployed, and release candidates can only be deployed with COPS approval 
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Tier One Testing 



To alleviate pressure on QRC testing EDG should establish tier one and 
tier two testing 
Tier One Testing: 

° Testing requirements for the specific environment in mind 
° Basic forensics that all tools need to pass 

A tool will remain a release candidate until tier one testing is complete 

It is up to EDG to ensure that tier one testing is inclusive enough to 
protect our tools 


AED and ESD: Establish a ' Chinese Menu’ of tests so that our customers can 
choose what is an immediate test and what is a test for later 
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The ERB 



The engineering review board’s role should be expanded to ensure a 
product meets requirements and is properly tested 
Current role of the ERB: 

° Ensure the tool meets requirements 
° Certify tool for delivery 
Expanded role of the ERB: 

° Live demo of the tool 
° Review tier one testing 
° Review automated test coverage 
° Ensure TPD has been filled in 
° Review of documentation 


AED and ESD: The ERB’s functionality should be expanded to review the above 
bullets 
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4ft 

Official Version 


An official version means that build can be deployed by the customer 
with no further interaction with EDG. Development doesn’t 
necessarily stop as new requirements could already be queued up. 
Tasks during this phase: 

° Operational use by the customer (hopefully) 

° Tier two testing 

° Long term regression testing (PSP and patch) 

° Generation of new requirements 

The tool can be deployed operationally without coordination with 
EDG 


AED and ESD: Rethink requirements so smaller official versions can be delivered 
more often. I.e. if a tool does three things no reason to hold up two of them 
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Tier Two Testing 


Tier two testing is meant to free up resources for QRC testing. Tests 
that fall under tier two are the “nice to haves” that bloat testing 
requirements and delay delivery of a tool 
Tier Two Testing Includes: 

° Other OS’s the tool may be deployed on 
° Other configurations the tool may be used with 
° Deeper forensic testing 

Issues found here don’t mean a new RC, they are submitted as a new 
requirement or DR. 


AED and ESD: This is a continuation of the ' Chinese Menu’ for testing and EDG 
should push for the nice to haves be moved to tier two 
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Version Control Via 
Bamboo ^ 


^ Philo 


Practice continuous integration (Cl) to 
automate building processes and 
maintain consistency in source base 
What is Cl? 

° Compile software on a standardized 
server (not personal workstations) 

° Manage different types of releases with 
build plans 
° Run automated tests to verify correct 
functionality ASAP (Fail Fast) 

° Track test execution results 
° All the above every time the source base 
changes 

Customize Bamboo to automate: 

plans for Release, Release Candidate, Evaluation Copy 
ESDe fy/n tegra tion with Dart for continuous testing 


Philosoraptor Cl ■ 

Continuous Integration for OSB tool Phil 


bugfix-PHILO-9-ensur 


feature-PH I LO-5-clean-up-on-target- 
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Continuous Testing 
Via DART 


Automate testing across multiple 
environments using DART 
° When applicable (rule of eight) 

° Execute unit/developer tests to verify 
functionality 

° Execute acceptance tests to verify 
requirements 

° Collect all test results into Bamboo for 
easy analysis 

A shared repository of dart tests shall 
be created to make testing easier (the 
EDG leafbag) 


0 Job: Run Developer Tests was successful 


Job Summary Tests Commits Artifacts Logs Metadata Issues 

Test results 

is 7 tests in total © < 1 second taken in total. 

Failed tests Successful tests (7) 

The following 7 tests have passed: 

All successful tests 


© DeployPayload BadPath © 

© DeployPayload GoodPath © 

© DeployPayload NullPath © 

© ExecutePayload BadPath © 

© ExecutePayload GoodPath © 

© ExecutePayload GoodPath_WithSpace © 
© ExecutePayload NullPath © 


AED: Create repeatable processes for automating tests via DART 
ESD: When possible recommend DART to development contracts 
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New Requirements ^ 


The tracking and creating of requirements remains the same for the 
most part 
Current process: 

° Draft requirement is generated 
° Accepted by the ERB 
° Tracked in IMIS (Soon to be ServiceNow) 

Recommended changes: 

° User stories should be captured in Jira 


AED: Ensure user stories are documented in Jira for the team to reference 
ESD: Ensure user stories passed along to the development team 
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Next Steps 


Incorporate final comments from branch chiefs 
Get official buy in from front office 
Use these four projects as examples 
0 CASCADE 
0 lOSTeam 
0 IMPROVISE 
0 PHILOSORAPTOR 

Get a branch into this cycle to use as an example 
0 Recommend OSB 

MOU with COG for the development cycle 

Brief EDG All-Hands with overview of this 

Brief branches two at a time to ensure maximum engagement 
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Task During This Phase: 

D Operation use by customer 
D Operational testing done on 
a continual basis 

□ Long term regression testing 
D Tier Two Testing (“Desired”) 
Movement To Next Phase 
Upon: 

D Creadon of DR 

0 Creation of new 

requirements 

□ Acceptance by the ERB 
Operationally Deployable? 

□ Deployable with traditional 




k 


k 


Task During This Phase: 

D Revaluation of tool and 
requirements by operators 
I Automated integration tests 
written by IV&V with the 
assistance of AED and 
previous testing scripts 
D Code reviews completed 
vt: Testing reviewed by COG 
and IV&V 

Movement To Next Phase 
Upon: 

ft Requirement 

documentation finalized 
1 Unit and integration testing 
completed and reviewed by 
both IV&V and branch chief 
D Branch chief approval 
Operationally Deployable? 

I Never - If COG desires 


use this operationally, a new 
requirements document 
must be generated. 



